EE 046746 - Technion - Computer Vision

Elias Nehme

Tutorial 07 - Homography, Alignment & Panoramas


Agenda


The largest panorama in the world (2014): Mont Blanc

In2WHITE Video

In2WHITE Full Image

Homographic usage examples:

http://blog.flickr.net/en/2010/01/27/a-look-into-the-past/

https://www.instagram.com/albumplusart/

Matching Local Features


Feature matching


Typical feature matching results


Parametric Transformations


Image Alignment


Given a set of matches, what parametric model describes a geometrically consistent transformation?

Basic 2D Transformations


Parametric (Global) Warping


Examples of parametric warps:

Basic 2D Transformations


Basic 2D Transformations - Translation


$$ \begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 & t_x\\ 0 & 1 & t_y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} x+t_x \\ y+t_y \\ 1 \end{bmatrix}$$

Basic 2D Transforation - Rotation


$$ \begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} cos(\theta) & -sin(\theta) & 0\\sin(\theta) & cos(\theta) & 0 \\0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$

Basic 2D Transformations – Translation and Rotation (2D rigid body motion)


$$ \begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} \cos(\theta) & -\sin(\theta) & t_x\\ \sin(\theta) & \cos(\theta) & t_y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$

Basic 2D Transformations – Scale


$$ \begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} s & 0 & 0\\ 0 & s & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$

Basic 2D Transformations – Similarity


Similarity transform (4 DoF) = translation + rotation + scale

Basic 2D Transformation - Aspect Ratio


$$ \begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix}= \begin{bmatrix} a & 0 & 0\\ 0 & \frac{1}{a} & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$

Basic 2D Transformations – Shear


$$ \begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} 1 & a & 0\\ b & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$

Basic 2D Transformations – Affine


Similarity transform (6 DoF) = translation + rotation + scale + aspect ratio +shear

Basic 2D Transformations – Projective - a.k.a - Homographic


$$ \begin{bmatrix} u \\ v \\ w \end{bmatrix} = \begin{bmatrix} h_1 & h_2 & h_3\\ h_4 & h_5 & h_6 \\ \color{red}{\text{h}}_\color{red}{\text{7}} & \color{red} {\text{h}}_\color{red}{\text{8}} & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} $$$$x' = u/w$$$$y' = v/w$$

Non-linear!

When do we get Homography?

Homography maps between:

For far away objects:

Computing Parametric Transformations


Computing Affine Transformation


$$ \begin{bmatrix} x_i' \\ y_i' \\ 1 \end{bmatrix}= \begin{bmatrix} h_1 & h_2 & h_3\\ h_4 & h_5 & h_6 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x_i \\ y_i \\ 1 \end{bmatrix}$$$$ \begin{bmatrix} \dots \\ x_i' \\ y_i' \\ \dots \end{bmatrix}= \begin{bmatrix} &&\dots\\ x_i & y_i & 1 & 0 & 0 & 0 \\ 0&0&0&x_i&y_i&1 \\ &&\dots \end{bmatrix} \begin{bmatrix} h_1\\h_2\\h_3\\ h_4 \\ h_5 \\ h_6 \\ \end{bmatrix} $$$$b = Ah$$

A_inv = pinv(A)

h = np.linalg.pinv(A)@b

$$ H = \begin{bmatrix} h_1 & h_2 & h_3\\ h_4 & h_5 & h_6 \\ 0 & 0 & 1 \end{bmatrix} $$$$x'_{new} = Hx_{new}$$

Computing Projective Transformation


$$ \begin{bmatrix} u_i \\ v_i \\ w_i \end{bmatrix}= \begin{bmatrix} h_1 & h_2 & h_3\\ h_4 & h_5 & h_6 \\ h_7 & h_8 & 1 \end{bmatrix} \begin{bmatrix} x_i \\ y_i \\ 1 \end{bmatrix} $$$$ x_i' = u_i/w_i$$

$$y_i' = v_i/w_i$$ $$ x_i' = \frac{h_1x_i + h_2 y_i + h_3}{h_7x_i+h_8y_i+h_9}$$

$$y_i' = \frac{h_4x_i + h_5 y_i + h_6}{h_7x_i+h_8y_i+h_9}$$
$$ \begin{bmatrix} &&&&&\dots\\ x_i & y_i & 1 & 0 & 0 & 0 & -x_ix_i'&-y_ix_i'& -x_i' \\ 0&0&0&x_i&y_i&1 & -x_iy_i'&-y_iy_i'& -y_i' \\ &&&&&\dots \end{bmatrix} \begin{bmatrix} h_1\\h_2\\h_3\\ h_4 \\ h_5 \\ h_6 \\ h_7\\h_8\\h_9\\ \end{bmatrix} = \begin{bmatrix} \dots \\ 0 \\ 0 \\ \dots \end{bmatrix}$$ $$Ah=0$$

where A is full rank. We are obviously not interested in the trivial solution $h=0$ hence we add the constraint $$||h||=1$$

$$ arg\min_h{||Ah||_2^2} \text{, } s.t ||h||_2^2=1 $$

Compute Projective transformation using SVD: $$arg \min_h{||Ah||_2^2} \text{, } s.t ||h||_2^2=1 $$

$$ ||UDV^Th||=||DV^Th||$$$$ ||V^Th||=||h|| $$

Hence, we get the following minimization problem:

$$ arg \min_h||DV^Th|| \text{ s.t. } ||V^Th||=1 $$

In Python:

(U,D,Vh) = np.linalg.svd(A,False)

h = Vh.T[:,-1]

Some more options to find $h$:

RANSAC


The RANSAC algorithm is extremly simple, but it often

Many improved algorithms:

Cool application 1: Planting images into other images

Cool application 2: BirdEye

Cool application 3: Looking into the past

Panorama


Obtain a wider angle view by combining multiple images

Warp - What we need to solve?


Forward Warping


Inverse Warping


Bilinear Interpolation

Sampling at $f(x,y)$:

$$ f(x,y) = (1-a)(1-b) f[i,j]\\ + a(1-b) f[i+1,j]\\ + ab f[i+1,j+1]\\ +(1-a)b f[i,j+1]\\ $$

Python:

Image Blending


Alpha Blending

Pyramid Blending:

  1. Build a Gaussian pyramid for each image
  2. Build the Laplacian pyramid for each image
  3. Decide/find the blending border (in the example: left half belongs to image 1, and right half to image 2 -> the blending border is cols/2)
    • Split by index, or
    • Split using a 2 masks (can be weighted masks)
  4. Constract a new mixed pyramid - mix each level seperatly acording to (3)
  5. Reconstract a blend image from the mixed pyramid
Blending Improvments

Panorama - Summary


Transformations in Deep Learning


Kornia - Computer Vision Library for PyTorch


Recommended Videos


Warning!

Video By Subject

Credits